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Abstract 

Objectives: Despite consensus regarding the conceptual foundation of crowding, and increasing 
research on factors and outcomes associated with crowding, there is no criterion standard measure of 
crowding. The objective was to conduct a systematic review of crowding measures and compare them 
in conceptual foundation and validity. 

Methods: This was a systematic, comprehensive review of four medical and health care citation databases 
to identify studies related to crowding in the emergency department (ED). Publications that "describe 
the theory, development implementation, evaluation, or any other aspect of a 'crowding measure- 
ment/definition' instrument (qualitative or quantitative)" were included. A "measurement/definition" 
instrument is anything that assigns a value to the phenomenon of crowding in the ED. Data collected 
from papers meeting inclusion criteria were: study design, objective, crowding measure, and evidence of 
validity. All measures were categorized into five measure types (clinician opinion, input factors, through- 
put factors, output factors, and multidimensional scales). All measures were then indexed to six valida- 
tion criteria (clinician opinion, ambulance diversion, left without being seen (LWBS), times to care, 
forecasting or predictions of future crowding, and other). 

Results: There were 2,660 papers identified by databases; 46 of these papers met inclusion criteria, were 
original research studies, and were abstracted by reviewers. A total of 71 unique crowding measures 
were identified. The least commonly used type of crowding measure was clinician opinion, and the most 
commonly used were numerical counts (number or percentage) of patients and process times associated 
with patient care. Many measures had moderate to good correlation with validation criteria. 
Conclusions: Time intervals and patient counts are emerging as the most promising tools for measuring 
flow and nonflow (i.e., crowding), respectively. Standardized definitions of time intervals (flow) and 
numerical counts (nonflow) will assist with validation of these metrics across multiple sites and clarify 
which options emerge as the metrics of choice in this "crowded" field of measures. 
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Crowding is a frequent and pervasive phenome- 
non for the majority of emergency departments 
(EDs) in the United States and around the 
world. 1 Crowding occurs when demand for services out- 
strips available resources. Recent studies have demon- 
strated that ED crowding is worsening in the US as 
demonstrated by longer waiting times to see clinicians 
and is likely exacerbated by the worsening problem of 
ED boarding, where admitted patients often stay for long 
periods in the ED waiting for inpatient bed placement. 2 ' 3 
Crowding adversely affects clinical outcomes, including 
mortality, and leads to delays in care for time-sensitive 
conditions, patient dissatisfaction with emergency care, 
and higher left without being seen (LWBS) rates. 4 

The most widely accepted conceptual framework of 
crowding is the input-throughput-output model. 5 Input 
factors are related to the demand for ED services, 
throughput factors are related to the ED processes of 
evaluation and treatment, and output factors are related 
to ED disposition. Despite consensus regarding the con- 
ceptual foundation of crowding, and increasing research 
focused on factors and outcomes associated with crowd- 
ing, there is no widely accepted way to measure crowd- 
ing. A review of the medical literature over 6 years ago 
demonstrated that among the numerous studies pub- 
lished on crowding, a common definition or measure of 
crowding did not exist. 6 Identifying metrics that are fea- 
sible, accurate, and reproducible may enable clinicians, 
administrators, researchers, and policy makers to better 
understand and manage ED crowding. The purpose of 
this study was to conduct a systematic review of all exist- 
ing crowding measures and compare them in terms of 
their conceptual foundations and validity. 



METHODS 



Study Design 

We conducted a systematic, comprehensive review of 
four medical and health care citation databases to iden- 
tify all studies related to crowding in the ED. Three 
independent reviewers screened the titles and abstracts 
from the comprehensive search and selected all studies 
that focused on the development or validation of a mea- 
sure(s) of crowding in the ED. A second set of review- 
ers scanned the full-text versions of the papers to verify 
that the study proposed and/or examined the validity of 
a crowding measure. Finally, a third set of reviewers 
examined all of the crowding measures in the eligible 
papers and summarized the evidence of their validity. 

Search Strategy 

In collaboration with a medical librarian (CC) from the 
Welch Medical Library at the Johns Hopkins University 
School of Medicine, we developed and executed a sys- 
tematic search strategy that allowed us to conduct a 
comprehensive literature review by separately searching 
PubMed (MEDLINE), CINAHL, Embase, and the 
Cochrane Database of Systematic Reviews (Cochrane 
Collaboration) to identify all scientific articles that were 
published or available on-line between January 1, 1966, 
and September 22, 2009, and related to the concepts of 
"emergency department" AND "crowding." The search 



strategy we used for PubMed, CINAHL, and Embase 
included creating database-specific queries that included 
official controlled vocabulary terms for each relevant 
concept when available, as well as keywords and key- 
word phrases for each concept (see Table 1). Controlled 
vocabulary terms were incorporated into the search 
queries for three of the databases: Medical Subject 
Headings [MeSH] for MEDLINE, CINHAL Headings for 
CINHAL, and Emtree terms for Embase. For the Cochra- 
ne Database of Systematic Reviews, only keywords and 
keyword phrases were used in the query; they are suffi- 
cient for comprehensive retrieval within this resource 
given its limited total number of records (see Table 1). 

The titles and abstracts of all papers related to 
crowding in the ED were reviewed to determine 
whether the study included a measure of crowding. 
Unless otherwise specified, scale, measure, and defini- 
tion were regarded as synonymous and are referred to 
simply as "measure" throughout this paper. Although 
the search strategy included the key word "surge" or 
key word phrases that included "surge," the studies 
were only considered eligible if "surge" described daily 
operational surge; papers related to disaster or mass 
casualty surge were excluded. Papers not published in 
English were also excluded. 

Sample Derivation 

Three independent reviewers (PC, SE, UH) completed 
an initial screen of all titles and available abstracts from 
the comprehensive search using the following inclusion 
criteria: papers with titles or abstracts that describe the 
theory, development, implementation, evaluation, or 
any other aspect of a "crowding measurement/defini- 
tion" instrument (qualitative or quantitative) and a 
"measurement/definition" instrument is anything that 
involves the act or process of assigning a value to the 
phenomenon of crowding in the ED. 

Following the same inclusion criteria, a second set of 
reviewers (MM, NR, RS) independently evaluated the 
full text versions of the papers and judged whether 
each study was appropriate for inclusion. Studies were 
included if at least two out of the three reviewers 
deemed them eligible. The reference sections of these 
full-text papers were also examined to ensure publica- 
tions were not missed during the comprehensive litera- 
ture search of the four databases. 

The final set of papers, having majority agreement 
for inclusion from the first and second screeners, were 
reviewed for information on ED crowding measures. 
A data abstraction instrument was developed (DA, CF, 
DH, UH, MM, JP, RS) to record information about the 
final set of papers. Out of a pool of six investigators 
(DA, CF, DH, JP, NR, FZ), three randomly assigned 
reviewers abstracted descriptive information from each 
publication. Publications that involved an identified or 
self-declared reviewer's conflict of interest were reas- 
signed. The abstraction instrument included categories 
concerning the study design, objective, crowding mea- 
sure, and evidence of validity. 

Based on data abstracted by the independent 
reviewers, two additional reviewers (UH and MM) 
reviewed all of the articles abstracted. Through confer- 
ence calls and threaded e-mail discussions, these two 
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Table 1 

Search Strategy Used for Each Database 





ED Concept 




Crowding Concept 






Controlled Vocabulary Term 


Key Words 


Controlled Vocabulary Term 


Key Words 


PubMed 


Emergency Service, 
Hospital [MeSH] OR 
Emergency Medical 
Services [MeSH] OR 
Emergency Medicine 
[MeSH] 


"emergency 
department" OR 
"emergency room" 
OR "ER" 


Crowding [MeSH] OR 
Surge Capacity 
[MeSH] 


crowd* OR overcrowd 
*"surge capacity" OR 
surge OR surges OR 
diversion OR 
occupancy OR 
congestion 


CINAHL 


Emergency Service+ 
OR Emergency 
Medical Services+ 
OR Emergency 
Medicine 


"emergency 
department" OR 
"emergency room" 
OR "ER" 


Crowding 


crowd* OR "surge 
capacity" OR surge 
OR surges OR 
overcrowd* OR 
diversion OR 
occupancy OR 
congestion 


EMBASE 


emergency ward/exp 
OR emergency health 
service/exp OR 
emergency 
medicine/exp 


"emergency ward" 
OR "emergency 
health service" OR 
"emergency 
medicine" OR 

"emergency 
department" OR 
"emergency room" 
OR ER OR 

"emergency medical 

"emergency service, 
hospital" 


crowding/exp OR 
hospital bed 
utilization/exp 


crowding OR "surge 
capacity" OR crowd* 
OR overcrowd OR 
surge OR surges OR 
diversion OR 
"hospital bed 
utilization" OR "bed 
utilization" OR 
occupancy OR 
congestion OR "bed 


Cochrane 
Database of 
Systematic 
Reviews 


N/A 


"emergency 
department" OR 
"emergency room" 
OR "emergency 


N/A 


crowd OR crowding 
OR overcrowding OR 
crowds OR diversion 



reviewers discussed areas of disagreement, and con- 
sensus was reached on categorization of all the mea- 
sures into one of five measure types (clinician opinion, 
input factors, throughput factors, output factors, or 
multidimensional scales), and indexed these to six com- 
parative validation criteria (clinician opinion-consensus 
panel, ambulance diversion, LWBS rate, times to care, 
forecasting [predicting future levels of crowding], and 
other [i.e., mortality, risk of methicillin-resistant Staphy- 
lococcus aureus infection, or opportunity loss of treat- 
ment capacity]). 

RESULTS 

A total of 2,660 papers were identified using the 
described databases and terms, of which 747 were 
unique papers that focused on measuring crowding in 
the title and abstract. After the initial screening, 92 
titles and abstracts met inclusion criteria for addressing 
the development or validation of measures of crowding 
in the ED. After the second screening of the full-text 
version of these publications, a total of 70 met inclusion 
criteria for data abstraction (i.e., after review of the 
full-text version of the publication, 22 papers were 
found to not address the development or validation of 
measures of ED crowding). Of the 70 publications that 
were reviewed, 46 were original research papers (see 
Figure 1). Upon review of the full-text versions of the 




Figure 1. Flow diagram of crowding in the ED measures 
review process. 

papers, 24 of the 70 were excluded for the following 
reasons: five were concept papers, 5,7-10 five were edito- 
rials, 11-15 three were reports, 16-18 six were review 
papers, 19-24 one was a website for an ED crowding 
measure calculator, 25 one was a performance stan- 
dard, 26 one was an abstract, 27 and two were not avail- 
able in English. 28 ' 29 

The 46 original studies contained 71 unique crowding 
measures. These measures were categorized into the 
five types of measures and indexed to their respective 
comparative validation criteria. Of the types of mea- 
sures, there are three clinician opinion measures, 17 
input measures, 21 throughput measures, 21 output 
measures, and nine multidimensional measures. A sum- 
mary of the results is provided in Table 2. 31-74 

Prevalence of Measures 

Clinician opinion, or perception of ED crowding, was 
the least commonly used type of crowding measure. 
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Table 2 

Measures of Crowding in the ED and Their Relationship to Validation Criteria 













Measure 
Type 


Measure 


Clinician 

Consensus 

Panel 
(Reference 
Number[s]) 


Ambulance 
Diversion 
(Reference 
Number[s]) 


LWBS 
(Reference 
Number[s]) 


Clinician 


Physicians feel rushed 31,32 
Clinician opinion of crowding 30 
EP satisfaction 47 
Waiting time 31-36 


31, 32 

47 
31, 32 


35 


30 




Waiting room filled > 6 hours/day 31,32 
Time to physician 37,47,57 


' 47 




57 




No. of arrivals 38 " 41 




41 


38 




No. of pts in waiting room 31 " 36,42 " 44 


31, 32, 42 


35 






No. of pts registered 35 ' 42 


42 


35 






No. or % of ambulance pts registered 45 
No. of pts awaiting triage 42 
No. of low-complexity pts 46 
No. of pts at each acuity level 37 
Average triage acuity level 45 
No. of new pts by usual care 45 
Percentage of open appointments in ambulatory 
care clinics 45 
LWBS 35 ' 37 


45 
42 

45 
45 
45 










35 




Throughput 


Average or % of pts who leave without treatment 

Ambulance diversion episodes 45 ' 69 
Average EMS waiting time 45 
ED beds at capacity > 6 hours or hallways filled > 
6 hours 31 ' 32 

Percentage of time ED > stated capacity 47 

No. of full rooms 42 

Total no. of pts in ED 37 ' 40 ' 42 ' 43 ' 47 " 52 


45 

' 45 
31, 32 

47 

42, 47, 48 


43 


43 




ED occupancy ra t e 33 ' 34 ' 45 ' 47 ' 53 " 55 


45, 47 


53, 54 


53 




No. of hallway pts 42 

No. of resuscitations in past 4 hours 43 

No. of pts being treated 36,44 


42 


43 


43 




No. of pts waiting for specialty consult or 
disposition by consultant > 4 hours 43 
No. of ED diagnostic orders 40 




43 


43 




No. of pts waiting test results 36 

No. of nurses working 43 

Pts treated by acuity per bed hours 45 

No. of pts per nurse or physician 45-48 

No. of pts admitted or discharged per physician 45 

Sum of pt care time per shift 56 

ED ancillary service turnaround time 42,45 


45 
45, 48 
45 

42, 45 


43 


43 
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Validation 
Criteria 



Forecasting 
(Reference 
Number[s]) Number[s]) 



Comments 



Significantly associated with clinician opinion of crowding. 31,32 
Significantly associated with LWBS. 30 

Significantly associated with clinician opinion of crowding. 47 
Not significantly associated with diversion 35 Two-hour-ahead 

forecasts good but relatively poor by 8 hours. 33 ' 34 
Significantly associated with clinician opinion of crowding 31 ' 32 
Significantly associated with clinician opinion of crowding, 47 an 

LWBS (did not wait). 57 

No validation, measured trends over time. 37 
Not associated with diversion. 41 

Significantly associated with LWBS, 38 wait time, boarding time, 
ED LOS. 39 



Lead indicator of ED a 
Significantly associate 
diversion, 35 waiting n 
Two- and 8-hour-aheai 



isus and d 
with clinic 



n of crowding, ' 



nd ED LOS. 44 
forecasts good to moderate, respectively.^ 
with clinician opinion of crowding, 42 but n 



Significantly associated with clinician opinion of crowding. 
Significantly associated with clinician opinion of crowding. 41 
Negligible increase in time to physician or ED LOS. 46 
No validation, measured trends over time. 37 
Significantly associated with clinician opinio 
Significantly associated with clinician opini 
Significantly associated with clinician opini 



n of crowding. 4i 
n of crowding. 4i 
n of crowding. 4i 



Not a: 



ciated with diversion. 



No validation, measured trer 






Significantly associated with 


clinici 




Significantly associated with 


clinici 


an opin 


Significantly associated with 






Significantly associated with 


clinici 




Significantly associated with 


clinici 




Significantly associated with 


clinici 




Significantly associated with 


clinici 




No validation, measured trer 






census and diagnostic reso 






Not associated with daily me 


an ED LOS, 



on of crowding. 45 ' 
on of crowding. 45 
on of crowding. 31-32 

on of crowding. 47 
on of crowding. 42 
on of crowding. 42 ' 47,48 
" Lead indicator of ED 

nor daily median ED 



Not a: 
ED c 



ciated with diversion or LWBS 



nificantly associated with clinician opinion of crowding. 
Good discriminator of diversion 53,54 and LWBS. 53 
Good forecasts 2, 8, and 12 hours ahead. 33,34,55 
Significantly associated with clinician opinion of crowding. 
Not associated with diversion or LWBS. 43 
Significantly associated with waiting room time 36,44 

Significantly associated with boarding time but not diversion 
or LWBS. 43 

Short-term forecasts poor but a significant predictor of future 

ED census. 40 
Significantly associated with waiting room time. 36 
No relationship noted with diversion or LWBS. 43 
Significantly associated with clinician opinion of crowding. 45 
Significantly associated with clinician opinion of crowding. 45,48 
Significantly associated with clinician opinion of crowding. 45 
Significantly associated with mortality within 10 days of ED vis 
Only one of the two studies associated with clinician opinion o 

crowding. 45 
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o Validation Criteria 



Boarding time components 45 
Observation unit census 45 

No. of pts waiting discharge ambulance pick-up 36 
ED admission transfer rate 45 
Hospital admission source 45 
Inpatient occupancy level 38 " 40-45,49 



Clinician 
Opinion, 
Consensus 

Panel 
(Reference 
Number[s]) 



Ambulance 
Diversion 
(Reference 
Number[s]) 



LWBS 
(Reference 
Number[s]) 



42, 45, 47 35, 41 



42, 45, 47, 58 



Hospital supply/demand forecast 45 
ED volume/inpatient bed capacity 45 
No. of inpatients ready for discharge 45 
Number of staffed acute care beds 47 
Inpatient processing times 45 
Inpatient laboratory, radiology, CT orders 40 
Time from request to bed assignment 47 
Time from bed ready to ward transfer 47 
Agency nursing expenditures 45 
Local home care service availability 69 
Alternate level of care bed availability 69 
Nearby EDs diverting ambulances 69 
EDWIN 53 - 54 ' 60 - 66-68 

NEDOCS 54 - 60 " 65 

Pediatric NEDOCS (PEDOCS) 64 

READI 54 - 65 - 67 

EDCS 67 

ED Work Score 54 - 71 
Critical Bed Status (CBS) 72 

System complexity 73 
Overcrowding Hazard Scale 74 



CT = computed tomography; EDWIN = ED Work Index; EMS = emergency medical services; EP = emergency physician; LOS = length 
of stay; LWBS = leave without being seen; MRSA = methicillin-resistant S. aureus; NEDOCS = National ED Over Crowding Study; 
PEDOCS = Pediatric ED Over Crowding Study; pt(s) = patient(s); READI = Real-time Emergency Analysis of Demand Indicators. 



ACAD EMERG MED • May 2011, Vol. 18, No. 5 • www.aemj.org 



Validation 
Criteria 



to Care 
(Reference 
Number[s]) 



Comments 



opinion of crowding. 45 



opinion of crowding. 4i 



No validation, measured trends 
Significantly associated with clii 
No validation, measured trends 
Not associated with diversion. 41 
Significantly associated with clir 
Not associated with diversion. 35 
Good forecasts 2 and 8 hours ahead. 33,34 
Significantly associated with LWBS, 38 waiting 

LOS. 39 ' 49,50 Poor short-term forecasts. 70 
Significantly associated with clinician opinion 

diversion, 35 ' 41 LWBS, 38 waiting room time, 36- ' 

boarding time, 39,44 and ED LOS 39 Not associate 

mortality or MRSA infections. 57 Good forecasts at 2 and 8 hours 

ahead. 33,34 

Significantly associated with clinician opinion of crowding, 42,45,47,5 ' 
diversion, 43 LWBS, 43 and opportunity loss of treatment capacity. 5 * 
Not associated with hospital mortality or MRSA infections. 57 
Short-term forecasts good but underestimated boarding time 33,34 



nificantly associated with cli 
Significantly associated with clii 
Significantly associated with no 
Significantly associated with cli 
Significantly associated with clii 
Significantly associated with clinic 
treatment time, 39 boarding time/ 
predictor of future ED census. 40 
Significantly associated with clinii 
Significantly associated with clinii 
Significantly associated with clinii 
Significantly associated with clinii 
Significantly associated with clinii 
Not a significant predictor of futu 
Significantly associated with clinii 
Significantly associated with clinii 
Significantly associated with clinii 
Significantly associated with clinii 
Significantly associated with clinii 
Significantly associated with clinii 
Significantly associated with clii 



of crowding, 
ician opinion of crowding. 45 
of pts waiting to be seen. 36 
ician opinion of crowding. 45 
ician opinion of crowding. 45 
ician opinion of crowding, 45 LWBS, 38 
, 39 and ED LOS. 39,49 Not a significant 



n of crowding. 45 
n of crowding. 45 
n of crowding. 45 
n of crowding. 47 
n of crowding. 45 

n of crowding. 47 
n of crowding. 47 
n of crowding. 45 
n of crowding. 69 
n of crowding. 6S 
n of crowding. 6i 
n of crowding, 66 " 



opinion of crowding 61,64,65 
60 and LWBS. 62 
with clinician opinion of crowding but 
operational variables. 64 

with clinician opinion of crowding 65 



diversion, 53,54,60,66 and LWBS. 
Significantly associated with clinici; 
(except for one study 63 ), diversion, 
Significantly 
outperformed by t\ 
Moderate 67 to poor 
and not a good disi 
Not a good discrimii 
Significantly 
Significantly 
of care times." 
Not validated against any criteria. 73 

Significantly associated with mortality <2, 7, and 30 days of ED 



ir of clinician opinion of crowding. 67 
with ambulance diversion. 54,71 
with time to room but not other ED process 
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Four papers included three types of clinician opinion 
measures; only one evaluated this type of measure 
against an objective outcome (LWBS rate), 30 and the 
remaining were based on ED director survey or con- 
sensus opinion. 

Input measures ranged from waiting times, to num- 
ber or percentage of patients as they arrived to the ED, 
to patient severity and complexity (e.g. number of 
patients at each acuity level). Of these, the most com- 
monly described measures were numerical counts or 
percentage of patients (as arrivals, in the waiting room, 
at triage or registration, by acuity, etc.), which were 
studied against all six types of validation criteria in 16 
different papers. 31 ^ 6 

Throughput measures included from ED capacity 
measures, numerical counts, or percentages of patients 
in the ED at various stages of ED evaluation, patient 
care times, and ED length of stay (LOS). Of these, the 
most commonly used measures were total number 
of patients in the ED (in 12 papers), 36 ' 37 " 40 ' 42 ^ 4 ' 47-52 
ED occupancy rate (in seven papers), 33 ' 34 ' 45 ' 47 ' 53-55 
and times associated with patient care (nine 
papers). 33_35 ' 37,41 ' 42 ' 45, 47,56 

Output measures included hospital measures of 
numerical counts, mean values or percentages of 
admissions, patients boarding in the ED, hospital beds 
and census, and times of care to leave the ED. Of these, 
the most commonly used measures were number, per- 
centage, or mean number of boarders (in 13 papers) 
and boarding times (in nine p a p ers ). 33 < 34 < 42 ' 43 ' 45 ' 47 - 57 - 59 

Finally, of the multidimensional indices, the most 
frequently studied measures were the National ED 
Overcrowding Study (NEDOCS) scale (in seven 
papers) 54,60-65 and the Emergency Department Work 
Index (EDWIN) measure (in six papers); 53 ' 54 ' 60,66-68 both 
were validated against each other and outcomes of 
clinician opinion, ambulance diversion, and LWBS rates. 

Validation of Measures 

Only one clinician opinion measure was validated 
against objective criteria and showed a positive rela- 
tionship with LWBS rates. 30 The three most commonly 
proposed input measures were the total number of 
patients in the waiting room, waiting room time, and 
the total number of arrivals. Both the total number of 
patients in the waiting room and the total number of 
ED arrivals were positively correlated with ED process 
times such as waiting room time and ED LOS. Further- 
more, one study found the number of ED arrivals was 
an important leading indicator of future ED census and 
demand for diagnostic resources 40 

The most commonly proposed throughput measures 
of crowding were ED census (total number of patients 
in the ED), ED occupancy rate, and ED LOS. All three 
measures were correlated with clinician opinion of 
crowding. The ED occupancy rate was also positively 
associated with ambulance diversion and LWBS rates. 
When measured daily, however, one study did not find 
that daily ED census was a significant predictor of daily 
mean ED LOS. 50 

The number or percentage of ED admissions; the 
number, mean number, or percentage of boarders; 
boarding time; and inpatient occupancy levels were the 



most common output measures proposed. ED admis- 
ions, boarders, and inpatient occupancy levels were 
gnificantly correlated with ED process times in addi- 
on to clinician opinion of crowding, ambulance diver- 
ion, and LWBS. Short-term forecasts of the number of 
boarders (i.e., 2 and 8 hours ahead) were more reliable 
than short-term forecasts of boarding time. 33-34 

Of the multidimensional measures, EDWIN 66 and NE- 
DOCS 61 have demonstrated the most evidence of valid- 
ity in terms of their positive association with clinician 
opinion of crowding, ambulance diversion, and LWBS. 
However, in a 1-year study period at six EDs, investiga- 
tors found that the ED occupancy rate discriminated as 
well as EDWIN during hours when EDs were on ambu- 
lance diversion or had one or more patients LWBS. 53 
Similarly, investigators found that the ED occupancy 
rate discriminated ambulance diversion episodes as 
well as other multidimensional measures at their facility 
during an 8-week study period. 54 

DISCUSSION 

Considerable attention has been devoted to the phe- 
nomenon of crowding in EDs over the past decade. 
Numerous measures have been proposed and devel- 
oped, and there is growing consensus of the need for 
quantitative, objective crowding measures that can be 
used across multiple sites and that are feasible and 
reproducible. The objective of this study was to conduct 
a systematic review of all existing crowding measures 
and compare them in terms of their conceptual founda- 
tion and validity. Seventy-one unique measures of ED 
crowding were identified in the medical literature. The 
vast number and wide variability in the metrics 
of crowding reflect how challenging this common 
phenomenon is to measure. 

The results of this review suggest that time intervals 
and numerical counts are becoming the most promi- 
nent measures of crowding in the medical literature. 
Furthermore, these categories of metrics represent the 
divergence of crowding measurement into two separate 
but related phenomena: patient flow and nonflow. 
The patient flow category relies predominantly on time 
intervals (e.g., ED total LOS and boarding time). 
Although the patient flow metrics are more challenging 
to observe in real time, they appear to be more general- 
izable across sites. The second category of metrics, 
nonflow, addresses the traditional concept of ED 
crowding, predominantly through the use of numerical 
counts of patients (e.g., ED census, number of waiting 
room patients, and number of boarders). Based on this 
review, we believe that it is most intuitive to think of 
these metrics as measures of nonflow. The advantage 
of the numerical patient counts is that they are easier to 
observe in real time; they may be, however, more chal- 
lenging to generalize across multiple EDs. 

The use of simpler measures of crowding appears to 
have come full circle. Early studies included surveys of 
ED providers and simple measures of census and ED 
boarding. The next series of articles focused on the 
development of multidimensional scales using real-time 
census, staffing, patient acuity, and hospital variables. 
Unfortunately, the challenge of capturing ED-specific 
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variation across and within multiple EDs as they transi- 
tion between normal and crowded conditions was 
extremely difficult. This challenge may also be difficult 
to overcome with increasing sophistication of the 
scales. The common theme of the multidimensional 
scales includes the observation that they work well in 
the EDs where they were derived. There is need for evi- 
dence of their scalability across multiple EDs. The 
inability of all EDs to capture the many components of 
these complex scales, however, may be a disadvantage. 

From this review, we do not believe that complex ED 
work scales, even the extensively studied EDWIN 66 and 
NEDOCS, 61 will generalize well enough across diverse 
settings for them to serve as criterion standard mea- 
sures of ED crowding. We recommend that attempts to 
develop more complex multidimensional workload 
scales should instead turn and focus on further valida- 
tion and investigation of less complicated measures. 
The transition back to simple and objective measures of 
crowding may be more practical. These measures are 
easier to collect and conceptually are more reproduc- 
ible. While many of the more straightforward measures 
have greater reproducibility as objective metrics (e.g., 
number of patients, ED LOS), others may be less so 
because of their subjective nature and being site-spe- 
cific (e.g., physicians feeling rushed, critical bed status). 
We suggest that future efforts to study and develop 
crowding measures be directed at using time intervals 
(flow) and counts (nonflow). 

Numerical counts (nonflow) such as total ED census, 
waiting room patients, and the number of boarders will 
be useful measures, particularly for point-in-time "snap- 
shots" of conditions within a single ED. Real-time counts 
are better used to test hypotheses prospectively and are 
more amenable for gauging conditions over time within 
a single institution when new processes are imple- 
mented. It is unclear, however, how easily numerical 
counts will generalize across sites effectively. In compar- 
isons across sites, numerical counts will need to be 
expressed as a percentage (or quantiles) of a predicted 
count for each ED or as rates such as occupancy (census 
divided by number of standardized ED beds). The rela- 
tionship between time intervals (flow) and how these 
gauge processes within a system is conceptually differ- 
ent from what is measured by numerical counts (non- 
flow). Measures of flow may be more generalizable 
across sites and thus have increased utility for multisite 
comparisons. Flow measures, however, are typically ret- 
rospective in nature and may be better used to calculate 
associations and generate hypotheses. Many organiza- 
tions engaged in performance improvement projects rely 
on timed measures of patient throughput and key opera- 
tional turnaround times. 75-77 While time intervals are not 
considered a crowding metric by these agencies, 75-77 the 
ultimate goal remains measurement of "flow." 

Measures of flow and nonflow (crowding) are not 
mutually exclusive. Both concepts are worth measuring, 
not only in single sites, but also as comparisons across 
sites. Numerical counts (as a percentage of allocated 
resources) and process times are likely to be linked 
both as predictors of crowding and as outcomes of 
crowding. It is very likely they measure different 
aspects of the phenomenon of ED crowding. Use of 



both flow and nonflow metrics highlight the fact that 
the phenomenon of crowding is not shouldered by the 
ED alone, but is also dependent on hospital- or system- 
wide factors. Time interval performance measures of 
factors outside the ED such as diagnostic efficiency 
(e.g., laboratory and radiology turnaround times), con- 
sult times, operating room activity, and inpatient bed 
availability, help to complement factors within the ED, 
such as patient counts, to provide a more complete 
picture of both the causes and effects of crowding. Ulti- 
mately the usefulness of these measures will be deter- 
mined by the extent to which they inform the system 
that is being investigated and how well they translate 
across settings. Consequences of crowding relevant to 
patients, clinicians, researchers, administrators, and 
policy makers include clinical outcomes, patient safety, 
patient and staff satisfaction, and cost of care. 

Another important finding of this review was the 
diversity of metrics that were conceptually measuring 
the same thing. For example, ED census was also 
referred to as total number of patients registered or 
total number of patients in the ED. Some studies mea- 
sured total number of arrivals, 38-41 while others used 
number of patients in the waiting room, 31-36 ' 42-44 the 
number of patients registered, 35,42 or the number of 
patients at triage. 42 Other studies used ED patient pro- 
cess times 37,41,42 ' 45 or LOS. 33-35,45,47 On a practical level, 
each of these metrics measures something different. 
Theoretically, however, they are all input and through- 
put measures using the same unit: numbers or time 
intervals. We believe that a standardized approach, per- 
haps using more simple methods of both time intervals 
(flow) and patient counts (nonflow), would be extremely 
helpful. Standardization of measures would give clini- 
cians, researchers, administrators, and policy makers 
the ability to compare and contrast crowding using 
similar references and units. This would also support 
future studies with the measurement and interpretation 
of crowding both within and across multiple EDs. 

LIMITATIONS 



The heterogeneous nature of the ED crowding literature 
and studies of the factors causing and resulting from it 
(surrogate measures) may have resulted in misclassiflca- 
tion of papers, study objectives, and measures. System- 
atically reviewing and summarizing proposed crowding 
measures was difficult, even for a group of emergency 
physicians and researchers with significant experience 
in studying and managing crowding. There were often 
disparities in the interpretation of results and measures, 
as some references included for final review did not 
focus on the goal of developing an instrument to mea- 
sure crowding, but instead evaluated crowding mea- 
sures in terms of outcomes (e.g., LWBS rate, ED LOS, 
mortality), were pre-post-intervention studies designed 
to alleviated crowding in EDs, or were descriptive sur- 
veys about crowding itself. This review was limited to 
evaluating the conceptual foundations and evidence of 
validity of the different measures. It did not evaluate the 
measures in terms of their reliability or responsiveness. 
This was largely because of the paucity of data available 
on these two traits for the majority of measures. 
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CONCLUSIONS 



While there remains no objective criterion standard 
measure of crowding in the ED, a combination of time 
intervals and patient counts appears to be emerging as 
the most promising tools for measures of flow and 
non-flow (i.e., crowding), respectively. Crowding scales 
that use multiple flow and nonflow variables simulta- 
neously have been developed, but their validity has not 
been reproducible outside of the settings where they 
were derived. Attempts to create additional complex 
multidimensional scales are unlikely to overcome this 
issue. Standardized definitions of time intervals (flow) 
and numerical counts (nonflow) will assist with valida- 
tion of these metrics across multiple sites. Ultimately 
the usefulness of measures will be determined by the 
extent to which they inform priority outcomes for the 
system, such as clinical outcomes, patient safety, patient 
and staff satisfaction, and costs of care. The validity and 
feasibility of both flow and nonflow metrics will clarify 
which options emerge as the metrics of choice in this 
"crowded" field of measures. 
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